Skip to content

7. Observability

In Machine Learning Operations (MLOps), observability is the key to understanding the performance, behavior, and health of your models and infrastructure in production. It provides the necessary tools and practices to monitor model performance, detect issues proactively, and make data-driven decisions for system optimization. This chapter covers the essential pillars of observability, empowering you to build robust and reliable MLOps pipelines.

  • 7.0. Reproducibility: Ensure your machine learning experiments are fully reproducible with MLflow Projects, facilitating collaboration, verification, and innovation.
  • 7.1. Monitoring: Master the principles of model monitoring by tracking key performance metrics, identifying behavioral shifts, and evaluating performance with the MLflow Evaluate API and Evidently.
  • 7.2. Alerting: Design effective alerting systems with tools like Slack, Discord, Datadog, and PagerDuty to promptly notify stakeholders of critical issues in your models or infrastructure.
  • 7.3. Lineage: Gain clarity on your data and model origins by tracking their entire lifecycle and transformations using MLflow Datasets.
  • 7.4. Costs and KPIs: Learn to manage the costs of AI/ML workloads and align your projects with business objectives by defining and tracking Key Performance Indicators (KPIs) with MLflow Tracking.
  • 7.5. Explainability: Dive into Explainable AI (XAI) and use techniques like SHAP to interpret model predictions, build stakeholder trust, and ensure transparency.
  • 7.6. Infrastructure: Monitor your system's health by tracking resource usage and performance metrics with MLflow System Metrics to optimize both efficiency and cost.

Checkpoint: explain a run from its evidence

Choose a chapter 5 training run and its corresponding model-version evaluation run. Find its input dataset source and digest, code version, model signature, heldout metrics, and system measurements. Write a short model card describing the intended use, known input constraints, test period, and one failure condition worth alerting on. You are ready when each claim points to a stored measurement or a clearly stated assumption. If the dataset path now contains different bytes, which evidence helps identify the run and which extra retention mechanism would let you reconstruct it?